Summary

This PR revamps the entire flashblocks rpc layer design, similar to the previous Realtime-feature (RTA) cache capabilities on the since discontinued xlayer-erigon (refer to: https://github.com/okx/xlayer-erigon/tree/dev/zk/realtime).

Essentially, this revamp implements the full solution 2 mentioned in the upstream reth's issue discussion thread: paradigmxyz/reth#20491, which is a more elegant design to the complex op-reth's flashblock rpc layer that is tightly coupled to the underlying anchoring canonical chainstate.

In addition, the new design overhauls and optimizes the flashblocks sequence validation, which is encapsulates inside the new FlashblockSequenceValidator. It aligns the sequence validation logic to the highly optimized upstream reth's engine payload validator ([ref].

Motivations

The best design for allowing minimal latencies is to allow a full state accumulation from incoming flashblocks that are ahead of the canonical chainstate, while allowing for a hybrid sync approach that:

1. State accumulated from incoming flashblocks that are ahead of the canonical chainstate are stored entirely in memory, while maintaining the underlying canonical chainstate as an anchor.
2. Optimize default CL sync from the state accumulation through a pre-warming mechanism of already executed flashblocks sequence (skip payload re-validation entirely)
3. Optimize flashblocks sequence validation with current Reth's payload validator optimizations. This includes
   • State root strategies (sparse trie caching, parallel merklization, and default synchronous SR resolution)
   • Prewarming states mechanism
   • Asynchronous resolution of transaction root, receipt root and log bloom on block assembling
4. Support processing flashblocks sequence block access list for faster state executions
5. Leverage on current reth engine's changeset cache, for asynchronous deferred computing of trie data
6. Automatic flushing of flashblocks state layer and in-memory trie changesets (up till CHANGESET_CACHE_RETENTION_BLOCKS), once canonical chainstate catches up
7. State queries (pending and confirmed states, also supporting latest tags) on flashblocks rpc tries to read from both the flashblocks state cache and underlying canon provider
8. State queries handle edge cases automatically such as chain re-orgs, flashblocks re-orgs, and flashblocks state layer lagging behind canonical chainstate

Revamped flashblocks RPC design

1. Flashblock cache layer

• Adds the new FlashblockStateCache layer which includes both the pending state and confirm state caches
• Confirm state cache indexes and stores full block data (state and stateless) in moemory
• Pending state cache stores the current flashblocks sequence state received from the sequencer
• On cache misses on the flashblocks state cache, all state queries are proxied to the underlying canon provider
• To handle edge case where canon provider is ahead of flashblocks state cache and cache reorg has not yet been triggered, if state retrieval hits on both underlying canon provider and flashblocks cache, we prioritize canon provider as the source of truth
• Holds the flashblocks state layer's trie data changeset cache, used for sparse trie task computation for accelerated flashblocks sequence pending state computation

2. Merklization

• Merklization on flashblocks sequence validation has been optimized to use reth's latest release new sparse trie optimizations (feat: sparse trie as cache paradigmxyz/reth#21583)
• Since after using sparse trie, state root calculations are extremely fast (with deferred trie data changeset computations done asynchronously), the sequence validator will compute SR incrementally for payload sequence validation
• All state root strategies (state root task - sparse trie, parallel and default synchronous SR) are all supported inside the flashblocks sequence validator.

3. Execution optimizations for flashblocks sequence validation

• Hyper optimized with prewarming state mechanism
• Support for flashblocks BAL
• Pre-execution of txs to optimistically pre-warm state read cache, and incremental re-use of account cache reads on the provider layer during EVM execution
• Async compute and resolution of transaction root, receipt root and log bloom duringblock assembling

4. Incremental execution support

• The sequence validation also supports incremental flashblocks sequence validation by only executing the suffix/delta transactions of incoming flashblocks
• Trie updates are also incrementally computed - this is implemented using the flashblocks state cache that:
  • Maintains and updates the flashblocks state layer that is ahead of the canonical in-memory cache
  • Arc of the canonical in-memory cache
  • The underlying provider and the anchor hash
• After the suffix transactions are validated, the suffix execution results are merged after every incremental builds with the prefix execution results

5. In-memory overlay provider suppor

• When generating the provider for incremental builds, the flashblocks state cache generates an in-memory overlay provider
• Overlays the anchor hash's underlying db provider with the canonical in-memory chainstate + the flashblocks state cache layer
• Pending state's pre-state bundle is also injected to the state provider for incremental validation during EVM execution
• For SR calculations, the in-memory sparse trie is re-used on top of that current pending block hash, so as to re-use prefix trie nodes and only calculate suffix changes

6. Flashblocks eth rpc extension

• Adds the new flashblocks-eth rpc extension (and maintaining compatibility with upstream OpEthApi)
• Override all eth json-rpc APIs to support using the new flashblocks state cache layer and the underlying canon chainstate provider
• Adds support for previously missing flashblocks eth APIs:
  • eth_sendRawTransactionSync
  • eth_getRawTransactionByBlockHashAndIndex
  • eth_getRawTransactionByBlockNumberAndIndex

7. Remove sequence manager, revamp to RawCache

• Removes the sequence manager entirely and its overly complicated updating logic from the execution task completion thread (updating calculated SR near target flashblocks count and re-broadcasting the same sequence)
• Simplifying to use a raw cache, which accumulates the raw flashblocks received by the sequencer
• Raw cache maintains similar logic to get the next best build-args, while tracking the last fully executed flashblocks sequence index

8. Add handlers for flashblocks state update logic

• Add handle_flashblocks handler - handle receiving a raw flashblock, and updating the raw cache
• Add handle_flashblocks_state handler - handle spawning task execution for flashblocks sequence validation and updating the flashblocks state cache
• Add handle_canonical_block - logic to handle canonical chainstate updates to flush flashblocks state cache layer, and also handle extreme cases like chain reorgs, etc

Supported flashblocks eth APIs

Block apis
[✅] eth_blockNumber
[✅] eth_getBlockByNumber
[✅] eth_getBlockByHash
[✅] eth_getBlockReceipts
[✅] eth_getBlockTransactionCountByNumber
[✅] eth_getBlockTransactionCountByHash

Transaction apis
[✅] eth_getTransactionByHash
[✅] eth_getRawTransactionByHash
[✅] eth_getTransactionReceipt
[✅] eth_getTransactionByBlockHashAndIndex
[✅] eth_getTransactionByBlockNumberAndIndex
[✅] eth_getRawTransactionByBlockHashAndIndex
[✅] eth_getRawTransactionByBlockNumberAndIndex
[✅] eth_sendRawTransactionSync

State apis
[✅] eth_call
[✅] eth_estimateGas
[✅] eth_getBalance
[✅] eth_getTransactionCount
[✅] eth_getCode
[✅] eth_getStorageAt

TODO:
Support eth_getLogs in the future

Design flaws on op-reth flashblocks rpc

1. The flashblocks rpc layer is tightly coupled to the canonical chainstate height. Thus, although they have since added optimistic builds to advance the pending state, however confirmed chainstates (using "latest" tags) are still dependent on the canonical chainstate
2. Flashblocks consensus sync mode does not make sense since flashblocks finality is less probably than default blocks. Flashblocks consensus sync also cannot currently account for rollbacks
3. Lack of support or design consideration for hybrid sync approach - tightly coupled with reason 2. Using a hybrid sync approach (CL sync + flashblocks state updates) is ideal for flashblocks design. However, full payloads from CL driven sync are currently re-executed on the default payload validator on engine_newPayload (even though the flashblocks sequence had already executed and validated the exact same payload)
4. SR calculations are triggered based on configurable values instead of a more deterministic approach, which can be a point of failure on rpc consensus. Also with reth's sparse trie async task feature implemented on reth v1.11 onwards, incremental SR calculations on payload validation becomes much cheaper
5. No flashblocks BAL implemented

On subsequent reth version upgrades (todo):

1. Add perf(tasks): add LazyHandle<T>, use for hash post state paradigmxyz/reth#22347 fixes once reth version is upgraded
2. Add perf: share executed tx counter with prewarming paradigmxyz/reth#22647 optimizations to share tx index
3. refactor(engine): PayloadProcessor::spawn_state_root paradigmxyz/reth#22604

Todos:

1. Add support for flashblocks block access list for faster flashblocks sequence execution and validation